Evaluating Measurement and Structural Invariance via Multi-Group Analysis
Tommaso Feraco
Invalid Date
Outline
Theoretical Background
R code
A real case study
Regressions
References
A hot topic
Introduction
COMMENTS?
The importance of Measurement Invariance
Researchers often compare groups of subjects on psychological variables … assuming that the adopted instruments similarly measure the same latent constructs across groups
Despite its appeal, this assumption is often not justified and needs to be tested to make comparisons across groups valid and interpretable
The assessment of Measurement Invariance is a prerequisite for meaningful comparisons across groups (or across time for the same groups)
…but not everyone (fully) agrees: doi:10.1080/10705511.2023.2191292
Invariance of a Structural Equation Model
More generally, testing for Invariance allows us to evaluate to what extent a hypothesized Structural Equation Model (SEM) can be considered invariant (i.e., having the same parameters) across groups
(Some) Applications
Evaluation of the psychometric properties of a psychological test or of a theoretical model on different sub-groups
For example, assessment of invariance across:
- gender
- age group
- pathological state
- culture, ethnicity, nationality
\end {itemize} - Longitudinal factorial invariance: the invariance of corresponding parameters across time within a group
Assessing Invariance:\ The Multi-Group Analysis
The Multi-group analysis is the most widely used method to assess invariance of a SEM model
In this lesson we will focus on a particular class of SEM models:
The Confirmatory Factor Analysis (CFA) models - In particular, we will adopt The Multi-Group Confirmatory Factor Analysis (MG-CFA) approach
Assessing Invariance: The starting point
Assessing Invariance: The idea
In MG analysis we start from a baseline situation in which the hypothesized CFA model is simultaneously estimated on all groups. At the beginning, all structural parameters are free to vary across groups
Next, more restrictive models are built in which some parameters (e.g., factor loadings) are constrained to be invariant across groups
The comparison of increasingly restrictive models allows to evaluate the increasing level of invariance between groups
Invariance steps:
Configural Invariance: the structure of the latent variable(s) is the same across groups (\(g \ne g'\)) and/or over time (\(\xi_g = \xi_{g'}\))
Metric Invariance (or Weak Invariance): factor loadings are equivalent across groups and/or over time (\(\Lambda_g = \Lambda_{g'}\))
Scalar Invariance (or Strong Invariance): intercepts of observed variables are equivalent across groups and/or over time (\(\tau_g = \tau_{g'}\))
Strict Invariance (or Residual or Invariant Uniqueness Invariance): residual variances of observed exogenous variables are equivalent across groups and/or over time (\(\Theta_{\delta,g} = \Theta_{\delta,g'}\))
…
Scalar/Strong invariance is required for meaningful mean comparisons
Assessing Factorial Invariance: A step-by-step guide
- **Steps from 1 to 4:** *Measurement Invariance*
- **Steps from 5 to 7:** *Structural Invariance*
MG-CFA scheme (steps 1-4)
Step 0: Separate models for each group
Step 1: Configural invariance
Step 1: Configural (non-)invariance
Configural invariance means that the “form” of the models is the same in the groups of interest. Form entails both the number of latent variables and whether the loadings are non-zero to begin with.
Step 2: Metric invariance
Step 2: Metric (non-)invariance
Metric invariance means that for each item, the loading of the factor on the item is the same in the two groups (or, again more precisely, that we cannot reject the hypothesis that the loadings are the same).
Step 2: Metric (non-)invariance
The source of group differences does not come from the latent variable!
Step 3: Scalar invariance
Step 3: Scalar invariance
Scalar invariance means that for each item, the intercept is the same. This means that group differences in the item responses are fully accounted for by group differences in the latent construct.
Step 4: Invariance of observed residual variances
Residual invariance means that for each item, the residual variance—the variance of the ominous E pointing into the items—is the same. We can again phrase this statistically: if we regressed the item scores on the factor, then the variance of the remaining residual would be the same in the groups (i.e., there would be homoscedasticity).
Step 4: Invariance of observed residual variances
The thing about the residual is that it captures everything that’s not explained in the model, and explaining changes in the amount of unexplained things seems a bit futile. Residual invariance is often not tested because it’s not necessary for latent mean comparisons. It’s a bit of an anticlimactic level to end on.
Step 5: Invariance of latent variances
Step 6: Invariance of latent covariances
Step 7: Invariance of latent means
Evaluation of level of invariance
Analysis of fit indices of the considered invariance model (\(\chi^{2}\), \(RMSEA\), \(CFI\), \(NNFI\)):
A good fit supports the validity of invariance - Comparison between fit indices of the considered invariance model and a less restrictive model:
A marked worsening of fit indices indicates that the considered invariance model is too restrictive and thus must be rejected
Note: It is strongly recommended to make a comprehensive evaluation based on different fit indices, rather than on a single fit criterion
Some general indications on model comparison
Let:
- $MOD_{A}$ be a model of invariance taken as reference model
- $MOD_{B}$ be a more restrictive model of invariance than $MOD_{A}$
We will have:
- $\Delta_{\chi^{2}_{BA}} = \chi^{2}_{MOD_{B}} - \chi^{2}_{MOD_{A}} \sim \chi^{2}_{BA}$ with $df$ equal to $df_{Mod_{B}} - df_{Mod_{A}}$
If the \(p-value\) associated with \(\chi^{2}_{BA}\) is less than a critical \(\alpha\) value then \({MOD_{B}}\) must be rejected Always assume \(\alpha_{CRITICAL}=.05\) is not reasonable - \(\Delta_{CFI_{BA}} = CFI_{MOD_B}-CFI_{MOD_A}\) If \(\Delta_{CFI_{BA}} > -.01\) then \({MOD_{B}}\) can be accepted - \(\Delta_{BIC_{BA}} = BIC_{MOD_B}-BIC_{MOD_A}\) If ${BIC{BA}} < 0 $ then \({MOD_{B}}\) is more plausible than \({MOD_{A}}\)
Partial invariance
When model invariance is untenable at a certain level (scalar, metric, or residual) we can determine what specific indicator(s) contribute to the misfit. Partial invariance is when most but not all parameters are constrained to be invariant. If partial invariance exists at a given level for a model, there are a variety of ways to proceed:
1. Leave the non-invariant indicator variables in the model, but not constrain them to be invariant across groups, arguing that the invariant indicators are sufficient to establish comparability of the constructs.
2. Argue that the differences between indicator variables are small enough that they would not make a substantive difference and proceed with invariance constraints in place.
3. Remove the indicator variables that are not fully invariant, and then re-run the invariance assessment.
4. Conclude that because there is not full invariance, the indicator variables must be measuring different constructs across the groups and, therefore, not use the indicators.
How can we recognize the parameters to ``free’’?
Inspection of parameters estimated separately for each group
The more the same parameter differs among groups, the more it will be plausible to free it - Inspection of Modification indices
- The *Modification Index* of a constrained parameter indicates the extent to which the model could improve if the parameter would be left free to vary across groups
- In general, we start from a more restrictive model and free one by one the parameters with higher Modification indices... until we obtain invariance
Note: Parameters left to vary freely across groups must be interpreted based on relevant theory
CFA hypothesized model
The Data
Data are in the data frame ``d’’
str(d)
'data.frame': 589 obs. of 8 variables:
$ x1 : num 1.21 -1.34 1.05 1.02 2.51 ...
$ x2 : num 0.725 -0.927 -1.037 1.953 1.823 ...
$ x3 : num 1.153 -1.635 3.148 -0.183 2.449 ...
$ x4 : num 2.185 -1.788 -0.101 0.978 0.77 ...
$ x5 : num 0.668 -0.414 1.843 -1.994 -1.774 ...
$ x6 : num -1.0233 -0.4756 -0.0675 -0.1684 1.2472 ...
$ x7 : num 1.046 -0.213 1.572 -1.513 0.194 ...
$ Group: int 1 0 0 1 1 1 1 0 1 1 ...
The groups are composed by:
table(d$Group)
0 1
234 355
Model building in R
This is the reference theoretical model that we want to evaluate
m <-"f1 =~ x1 + x2 + x3 + x4f2 =~ x5 + x6 + x7"
Step 0: Separate models for each group
First of all, try to fit the model separately in the two groups.
This is not configural invariance! We are just testing if the model works in the two groups, not if the model has the same ‘form’ in the two groups.
# Fit the model in the two groupsm0<-cfa(m, data=d[d$Group==0,])m1<-cfa(m, data=d[d$Group==1,])# Evaluating parameters and goodness-of-fit indicessummary(m0,fit.measures=TRUE)summary(m1,fit.measures=TRUE)
Step 1: Configural invariance
Fitting a configural model is very easy: just add group=“groupVariableName” to the cfa sintax.
# Fit the configural modelm.conf<-cfa(m, data=d, group ="Group")# Evaluating parameters and goodness-of-fit indicessummary(m.conf)fitmeasures(m.conf, fit.measures = fi)# Do they decrease compared to the full model?
Step 2: Metric invariance
Adding metric invariance to the model means fixing the loadings to force them to be equal in the two groups.
# Fit the model for metric invariancem.metr<-cfa(m, data=d, group ="Group",group.equal ="loadings")# Evaluating parameters and goodness-of-fit indicessummary(m.metr)fitmeasures(m.metr, fit.measures = fi)# Comparison between metric and configural invariance (delta chi)anova(m.metr,m.conf)# Comparison between metric and configural invariance (delta CFI)fitMeasures(m.metr,"cfi") -fitMeasures(m.conf,"cfi")# Comparison between metric and configural invariance (delta BIC)fitMeasures(m.metr,"bic")-fitMeasures(m.conf,"bic")# Inspection of the modification indiceslavTestScore(m.metr)parameterTable(m.metr)
Steps 3-7: Evaluation of different invariance models
Through the option group.equal , it is possible to constrain groups of parameters to be equal across groups in order to assess increasingly restrictive invariance hypotheses:
Constrained parameters
In R
Factor loadings
loadings
Intercepts of manifest variables
intercepts
Residual variances of manifest variables
residuals
Residual covariances of manifest variables
residual.covariances
Residual variances of latent variables
lv.variances
Residual covariances of latent variable
lv.covariances
Intercepts/means of latent variables
means
All regression coefficients
regressions
# Example: Model for assessing scalar invariancem.scal<-cfa(m,d,group="Group",group.equal=c("loadings","intercepts"))
This is DEPRECATED from the authors of the package and will be deleted from future versions of semTools.
You can use it exploratorily, but you cannot:
follow and interpret the estimates step by step
model partial invariance!
Let’s see what partial invariance is.
Models of partial invariance
Through the option group.partial , you can test for partial invariance by allowing a few parameters to remain free:
# After the inspection of MI, we decided to estimate a model# of partial metric invariance in which the loadgin of the# item x5 is free to vary between groups:m.metr<-cfa(m,data=d,group="Group",group.equal="loadings",group.partial="f2 =~ x5")
What should we do now?
Should we compare this model with what?
How can we interpret the results?
… Now it’s time for a real case study!
The case study
*SEM work with variance-covariance matrices. For this reason, it is sufficient to find it in the original articles to use their “data”!. The data we will use have been generated based on the parameters provided in the article, modifying the sample size. How? Open dataGen.R* in the \texttt{'data' folder}
Theoretical model and reference groups
- Group A: Youths With Manics Symptoms ($n=150$)
- Group B: Control Group ($n=150$)
# Separate modelsm.man<-cfa(model,data=dmg[dmg$diagnosis=="manic",])m.nor<-cfa(model,data=dmg[dmg$diagnosis=="norming",])# Inspection of fit indicesfitMeasures(m.man,c("chisq","df","rmsea","cfi","nnfi"))
# Model of metric invariancem.metr<-cfa(model,dmg,group="diagnosis",group.equal="loadings")# Inspection of fit indicesfitMeasures(m.metr,c("chisq","df","rmsea","cfi","nnfi"))
# Model of Scalar invariancem.scal<-cfa(model,dmg,group="diagnosis",group.equal=c("loadings","intercepts"))# Inspection of fit indicesfitMeasures(m.scal,c("chisq","df","rmsea","cfi","nnfi"))
# From a first analysis, we can see that the intercept of the variable "Sim"# (constraint p21 == p50) has a high Modification Index# ... freeing this parameter results in an improved fit# (see parTable(m.scal), column id, to "find the meaning" of p21 and p50)# Let's build a model of Partial Metric invariance by freeing the intercept of "Sim" ...
Partial Scalar invariance (1)
m.scal.P=cfa(model,dmg,group="diagnosis",group.equal=c("loadings","intercepts"),group.partial="Sim~1")# Inspection of fit indicesfitMeasures(m.scal.P,c("chisq","df","rmsea","cfi","nnfi"))
# Partial Scalar invariance is satisfactory# and now becomes our reference model# Question: How can we interpret this result?
Invariance of Residuals of observed variables(1)
# Note: parameter "Sim~1" remains freem.rvo=cfa(model,dmg,group="diagnosis",group.equal=c("loadings","intercepts","residuals"),group.partial="Sim~1") #Note that this is still here# Inspection of fit indicesfitMeasures(m.rvo,c("chisq","df","rmsea","cfi","nnfi"))
# The Invariance of Residuals of observed variables# is not satisfactory. Let's take a look at equality constraints
Inspection of equality constraints
lavTestScore(m.rvo)$uni
# (... see complete output )# From a first analysis, we can see that# the residuals of variables “Comp” and “PicComp”# have Modification indices that are# particularly high, so let's free them
Invariance of Residuals of observed variables (1)
# Partial Invariance of Residuals of observed variablesm.rvo.P=cfa(model,dmg,group="diagnosis",group.equal=c("loadings","intercepts","residuals"),group.partial=c("Sim~1","PicComp~~PicComp","Comp~~Comp"))# Fit indicesfitMeasures(m.rvo.P,c("chisq","df","rmsea","cfi","nnfi"))
# Paartial Invariance of Residuals of observed variables# is satisfactory.# Question: What can we say about overall# Measurement Invariance of the baseline theoretical model?
Invariance of Variance of latent variables (1)
m.vvl=cfa(model,dmg,group="diagnosis",group.equal=c("loadings","intercepts","residuals","lv.variances"),group.partial=c("Sim~1","PicComp~~PicComp","Comp~~Comp"))# Fit indicesfitMeasures(m.vvl,c("chisq","df","rmsea","cfi","nnfi"))
m.cvl=cfa(model,dmg,group="diagnosis",group.equal=c("loadings","intercepts" ,"residuals","lv.variances","lv.covariances"),group.partial=c("Sim~1","PicComp~~PicComp","Comp~~Comp"))# Fit indicesfitMeasures(m.cvl,c("chisq","df","rmsea","cfi","nnfi"))
# last stepm.med=cfa(model,dmg,group="diagnosis",group.equal=c("loadings","intercepts" ,"residuals","lv.variances","lv.covariances","means"),group.partial=c("Sim~1","PicComp~~PicComp","Comp~~Comp"))# Fit indicesfitMeasures(m.med,c("chisq","df","rmsea","cfi","nnfi"))
# This last model is also satisfactory# ANY COMMENTS?
Summing up
Models
npar
df
chisq
cfi
tli
nnfi
agfi
srmr
rmsea
bic
aic
Manics
17
19
54.05
0.95
0.92
0.92
0.84
0.05
0.11
5590.65
5539.47
Norming
17
19
18.15
1
1
1
0.94
0.03
0
5718.64
5667.46
Configural
50
38
72.2
0.97
0.96
0.96
0.98
0.04
0.08
11424.11
11238.93
Metric
44
44
88.15
0.96
0.95
0.95
0.98
0.06
0.08
11405.84
11242.88
Scalar
38
50
144.67
0.92
0.91
0.91
0.97
0.08
0.11
11428.13
11287.39
Scalar Partial
39
49
100.17
0.96
0.95
0.95
0.98
0.07
0.08
11389.34
11244.9
Residual Variances
31
57
127.34
0.94
0.94
0.94
0.98
0.08
0.09
11370.88
11256.07
Residual Variances Partial
33
55
102.42
0.96
0.96
0.96
0.98
0.07
0.08
11357.37
11235.14
Latent Variances
31
57
105.83
0.96
0.96
0.96
0.98
0.09
0.08
11349.36
11234.55
Latent Covariances
30
58
106.41
0.96
0.96
0.96
0.98
0.09
0.07
11344.25
11233.13
Latent Means
28
60
110.93
0.96
0.96
0.96
0.98
0.09
0.08
11337.36
11233.66
Interpretations, comments, or questions?
To combine usefulness and fun …
Graphical representation of the Configural Invariance model with standardized parameters (attached R code)
Excercise: Graphically represent the former invariance models using the function semPaths of the package semPlot
Invariance of regression coefficients
Test of multigroup invariance can also be used to compare differences in the regression coefficients of two or more groups.
Invariance of regression coefficients
Following the same steps of the previous example, we can add the "regressions" to the group.equal argument.
fitPreg <-sem(path, d, group ="group",group.equal ="regressions")
You can apply this to a path model with manifest variables only or to a full SEM after measurement invariance is tested.
QUESTIONS? LET'S SEE THE "code"
Useful references
Beaujean, Freeman, Youngstrom & Carlson (2012). The structure of cognitive abilities in youths with manic symptoms: a factorial invariance study. Assessment, 19, 462 - 471
Non-invariance materials were directly taken from the wonderful blogpost of Julia Rohrer. READ IT!
How much is invariance disregarded?
Protzko’s humorous preprint on what we can actually claim with measurement invariance…without good measures!